Papers with annotation tool
NLP Lean Programming Framework: Developing NLP Applications More Effectively (N18-5)
Copied to clipboard
| Challenge: | NLPf is a framework for creating custom natural language processing models and pipelines by utilizing common software development build systems. |
| Approach: | They propose a framework for creating custom NLP models and pipelines by utilizing common software development build systems. |
| Outcome: | This framework allows developers to train and integrate domain-specific NLP pipelines into their applications seamlessly. |
SLATE: A Super-Lightweight Annotation Tool for Experts (P19-3)
Copied to clipboard
| Challenge: | a new annotation tool is designed to fill the niche of a lightweight interface for terminal users . current tools are built with direct manipulation via a Graphical User Interface (GUI) this approach is time-consuming and difficult to modify . |
| Approach: | They propose a terminal-based annotation tool that supports multiple annotations . they use a text-based interface that uses almost the entire screen to display documents . |
| Outcome: | The proposed tool is designed to fill the niche of a lightweight interface for users with a terminal-based workflow. |
LightTag: Text Annotation Platform (2021.emnlp-demo)
Copied to clipboard
| Challenge: | LightTag is a text annotation tool built on the premise of global optimization by addressing annotator as well as project managers and data scientists who manage the work and enforce production quality. |
| Approach: | They propose to use LightTag to optimize the global NLP process by addressing annotators as well as project managers and data scientists who manage the work and enforce production quality. |
| Outcome: | The proposed tool is based on the theory of constraints and is available for free for academic use. |
metaCAT: A Metadata-based Task-oriented Chatbot Annotation Tool (2020.aacl-demo)
Copied to clipboard
| Challenge: | Creating high-quality annotated dialogue corpora necessitates a high level of human engagements. |
| Approach: | They propose to develop an annotation tool specifically for developing task-oriented dialogue data that provides comprehensive metadata annotation coverage to the domain, intent, and span information. |
| Outcome: | The tool provides comprehensive metadata annotation coverage to domain, intent, and span information. |
Introducing Frege to Fillmore: A FrameNet Dataset that Captures both Sense and Reference (2022.lrec-1)
Copied to clipboard
| Challenge: | a widely supported claim in the fields of semantics and philosophy is that meaning arises from the combination of sense and reference. |
| Approach: | They propose a tool that facilitates both referential- and frame annotations of language-independent corpora. |
| Outcome: | The Dutch FrameNet annotation tool facilitates both referential- and frame annotations of language-independent corpora. |
ActiveAnno: General-Purpose Document-Level Annotation Tool with Active Learning Integration (2021.naacl-demos)
Copied to clipboard
| Challenge: | Existing tools for document-level annotation lack document-based quality and flexibility. |
| Approach: | a new annotation tool is being developed for industry and research use cases . a configurable user interface and a RESTful API are included . authors propose to use ACTIVEANNO as default for document-level annotation . |
| Outcome: | ACTIVEANNO is an annotation tool for industry and research use cases. |
TALEN: Tool for Annotation of Low-resource ENtities (P18-4)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a task that requires a large amount of training data and annotators who do not speak the language are hard or impossible to find. |
| Approach: | They propose a web-based interface for named entity annotation in low-resource settings . TALEN includes in-place lexicon integration, TF-IDF token statistics, Internet search, and entity propagation . |
| Outcome: | The proposed interface performs better than a popular annotation tool and is more accurate and recall-rich than the current one. |
APLenty: annotation tool for creating high-quality datasets using active and proactive learning (D18-2)
Copied to clipboard
| Challenge: | APLenty is an annotation tool for creating high-quality sequence labeling datasets using active and proactive learning. |
| Approach: | They present APLenty, an annotation tool for creating high-quality sequence labeling datasets using active and proactive learning. |
| Outcome: | The proposed tool is highly flexible and can be adapted to various other tasks. |
ViClaim: A Multilingual Multilabel Dataset for Automatic Claim Detection in Videos (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing efforts in misinformation detection focus on written text, leaving a significant gap in addressing the complexity of spoken text in video transcripts. |
| Approach: | They propose to annotate video transcripts in three languages and six topics using a custom annotation tool. |
| Outcome: | The proposed tool shows strong cross-validation performance but challenges for generalization to unseen topics. |
LIDA: Lightweight Interactive Dialogue Annotator (D19-3)
Copied to clipboard
| Challenge: | Dialogue systems are dependent on the quality of the data used to train them. |
| Approach: | They propose to develop an annotation tool specifically for conversation data that handles the entire dialogue annotation pipeline from raw text to structured conversation data. |
| Outcome: | The proposed tool handles the entire dialogue annotation pipeline from raw text to structured conversation data and has a dedicated interface to resolve inter-annotator disagreements. |
CDˆ2CR: Co-reference resolution across documents and domains (2021.eacl-main)
Copied to clipboard
| Challenge: | Current models for cross-document co-reference resolution assume that all documents are of the same type or fall under the same theme. |
| Approach: | They propose a task for cross-document cross-domain co-reference resolution (CD2CR) task aims to identify links between entities across heterogeneous document types. |
| Outcome: | The proposed task outperforms current state-of-the-art models on CD2CR in cross-domain, cross-document setting. |
Tools Impact on the Quality of Annotations for Chat Untangling (2021.acl-srw)
Copied to clipboard
| Challenge: | Creating annotated datasets is time-consuming and expensive . a tool that minimizes task load and facilitates the annotation process is lacking in studies on how it influences annotation quality. |
| Approach: | They propose to use SLATE and Parlay to improve annotation quality for a task of chat-untangling. |
| Outcome: | The proposed tool improves the user experience for the task of chatuntangling. |
ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution (2023.findings-eacl)
Copied to clipboard
Ankita Gupta, Marzena Karpinska, Wenlong Zhao, Kalpesh Krishna, Jack Merullo, Luke Yeh, Mohit Iyyer, Brendan O’Connor
| Challenge: | Existing datasets vary in definition of coreferences and are curated for linguistic experts. |
| Approach: | They propose to use ezCoref to create a crowdsourcing-friendly coreference annotation methodology that teaches annotators only cases that are treated similarly across existing datasets. |
| Outcome: | The proposed method reannotates 240 passages from seven existing english coreference datasets while teaching annotators only cases that are treated similarly across them. |
PAWLS: PDF Annotation With Labels and Structure (2021.acl-demo)
Copied to clipboard
| Challenge: | Existing tools for annotation of PDFs are limited to a web browser, allowing users to extract semantically meaningful regions from PDFs. |
| Approach: | They propose an annotation tool specifically designed for Adobe’s Portable Document Format (PDF) PAWLS supports span-based textual annotation, N-ary relations and freeform, non-textual bounding boxes. |
| Outcome: | The proposed tool supports span-based textual annotation, N-ary relations and freeform, non-textual bounding boxes. |
OpenRT: An Open-source Framework for Reasoning Over Tabular Data (2023.acl-demo)
Copied to clipboard
| Challenge: | Existing table pre-training methods are benchmarked on a limited number of datasets with varying configurations, resulting in a lack of unified, standardized, fair, and comprehensive comparison between methods. |
| Approach: | They propose to use OpenRT to reproduce existing table pre-training models and develop new models quickly. |
| Outcome: | The proposed framework reproduces existing table pre-training models and compares them against four question answering, one fact checking, and one faithful text generation datasets. |
Redcoat: A Collaborative Annotation Tool for Hierarchical Entity Typing (D19-3)
Copied to clipboard
| Challenge: | Recent advances in natural language processing (NLP) are fuelled by high quality annotated datasets. |
| Approach: | They introduce Redcoat, a web-based annotation tool that supports collaborative hierarchical entity typing. |
| Outcome: | The proposed annotation tool reduces the time it takes for project creators to set up and distribute projects to annotators and scales the workload depending on the number of active annotator. |
FAST: Fast Annotation tool for SmarT devices (2021.emnlp-demo)
Copied to clipboard
| Challenge: | In real-world applications, annotators with the same attributes are required to annotate whether the outputs of natural language generation systems are fluent or not. |
| Approach: | They propose an annotation tool for application tasks that focuses on the user experience of mobile devices and can be customized to fit various tasks. |
| Outcome: | The proposed tool can annotate faster than existing methods while maintaining the quality of annotation. |
A Multi-Task Transformer Model for Fine-grained Labelling of Chest X-Ray Reports (2024.lrec-main)
Copied to clipboard
| Challenge: | Chest radiography is one of the most commonly performed imaging examinations, with over 50 million chest X-rays annually in the United States . manual review of chest x-ray images can be onerous process prone to errors due to large volumes of images interpreted daily by radiologists . |
| Approach: | They propose a transformer-based model that segments radiology reports into semantically coherent segments and classifies each segment using a set of 37 radiological abnormalities. |
| Outcome: | The proposed model achieves state-of-the-art on report segmentation (0.0442 WinDiff) and multi-label classification (0.8 report-level macro F1) over 37 radiological labels and 8 NLP-specific labels. |
Exploring Amharic Sentiment Analysis from Social Media Texts: Building Annotation Tools and Classification Models (2020.coling-main)
Copied to clipboard
| Challenge: | Existing crowdsourcing platforms do not support sentiment analysis for Amharic, and there are no expert researchers in the area. |
| Approach: | They propose to build a social-network-friendly Amharic sentiment analysis tool using the Telegram bot and collect 9.4k tweets where each tweet is annotated by three Telegram users. |
| Outcome: | The proposed system outperforms existing classifiers in Amharic and other low-resource languages due to the widespread use of sarcasm and figurative speech. |
A Corpus of Drug Usage Guidelines Annotated with Type of Advice (L18-1)
Copied to clipboard
| Challenge: | Current research indicates patients are often unaware of such critical information / advice related to their prescription drugs due to lack of communication with their doctors and/or pharmacists. |
| Approach: | They propose an annotation scheme for annotating safety critical advice from drug usage guidelines and an annotated dataset containing drug usage guideline data. |
| Outcome: | The proposed dataset will accelerate further release of annotated drug usage guideline datasets and research on automatically filtering safety critical information from these documents. |
Fine-grained Semantic Textual Similarity for Serbian (L18-1)
Copied to clipboard
| Challenge: | Semantic textual similarity (STS) is a task of assigning a numerical score to short texts based on the level of semantic equivalence between them. |
| Approach: | They propose to annotate Serbian STS dataset with fine-grained similarity scores . they propose a supervised bag-of-words model that combines part-of speech weighting with term frequency weighting . |
| Outcome: | The proposed model outperforms existing models on the Serbian STS News Corpus . the proposed model is based on a new morphologically rich language . |
QUD-Based Annotation of Discourse Structure and Information Structure: Tool and Evaluation (L18-1)
Copied to clipboard
| Challenge: | a new annotation scheme and discourse-analytic method is developed for information structure annotation. |
| Approach: | They propose a new annotation scheme and a discourse-analytic method based on Questions under Discussion . they introduce a tool which enables the analyst to semi-automatically segment texts and enhance them with QUDs . |
| Outcome: | The proposed method achieves good inter-annotator scores and good agreement with discourse annotations. |
Modeling Factual Claims with Semantic Frames (2020.lrec-1)
Copied to clipboard
| Challenge: | In recent years, the proliferation of misinformation has reached a staggering pace eroding people's confidence in politics and even affected democracies. |
| Approach: | They propose an extension of the Berkeley FrameNet for the structured and semantic modeling of factual claims. |
| Outcome: | The proposed extension provides 2,540 fully annotated sentences and can be used to understand how these frames are intended to work and to train machine learning models. |
CantoMap: a Hong Kong Cantonese MapTask Corpus (2020.lrec-1)
Copied to clipboard
| Challenge: | a corpus of connected spoken Hong Kong Cantonese is constructed to study the phonology and semantics of the language. |
| Approach: | They propose to build a corpus of connected spoken Hong Kong Cantonese with phonemic transcription and controlled elicitation tasks. |
| Outcome: | The proposed corpus contains 768 minutes of recordings and transcripts of forty speakers. |
Don’t Annotate, but Validate: a Data-to-Text Method for Capturing Event Data (L18-1)
Copied to clipboard
| Challenge: | Existing methods to create event data are limited by ambiguity and variation in the data. |
| Approach: | They propose a method to obtain large volumes of text corpora with event data . they use a tool to annotate texts and enrich the reference texts with event coreference annotations. |
| Outcome: | The proposed method obtains large volumes of high-quality text corpora with event data . the data obtained with this method have high precision and at a large scale . |
Testing Focus and Non-at-issue Frameworks with a Question-under-Discussion-Annotated Corpus (2022.lrec-1)
Copied to clipboard
| Challenge: | Annotated German driving reports for question-under-discussion analysis are lacking in the literature on QUDs. |
| Approach: | They propose to annotate a German driving report corpus for QUD analysis . they show focus-related meaning aspects are essentially confirmed . |
| Outcome: | The annotated corpus of German driving reports shows that focus-related meaning aspects are essentially confirmed, indicating a sufficent accuracy of the annotations. |
Annotating Attribution Relations in Arabic (L18-1)
Copied to clipboard
| Challenge: | Current studies focus on using lexical terms in long texts to verify author identity. |
| Approach: | They propose to annotate attributed arguments to the source in Arabic news with required syntactical and semantic features with required features. |
| Outcome: | The proposed method is applied to Arabic news and is compared with existing tools and methods. |
Project PIAF: Building a Native French Question-Answering Dataset (2020.lrec-1)
Copied to clipboard
Rachel Keraron, Guillaume Lancrenon, Mathilde Bras, Frédéric Allary, Gilles Moyse, Thomas Scialom, Edmundo-Pavel Soriano-Morales, Jacopo Staiano
| Challenge: | a lack of data for non-English languages is limiting the development of downstream tasks such as Question Answering. |
| Approach: | They propose to collect a native French Question Answering Dataset using a participatory setup. |
| Outcome: | The proposed tool allows volunteers to participate in crowdsourced annotations in French. |
The Multimodal Annotation Software Tool (MAST) (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing visual and multimodal annotation systems are limited in scope and applicability. |
| Approach: | They propose a tool that allows users to analyze visual and multimodal documents . they aim to provide a powerful and innovative annotation tool with application across fields . |
| Outcome: | The MAST tool allows users to analyze visual and multimodal documents . it allows annotation theories to be citable, while evolving and being shared . |
ATC-ANNO: Semantic Annotation for Air Traffic Control with Assistive Auto-Annotation (2020.lrec-1)
Copied to clipboard
| Challenge: | ATC communications are a challenging domain for automatic speech recognition (ASR) due to the time-sensitive nature of their task, annotators must have prior experience with ATC communication. |
| Approach: | They propose a tool for the transcription and semantic annotation of air traffic communications. |
| Outcome: | The proposed tool can annotate four times as many utterances in a single time. |
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers (2024.findings-emnlp)
Copied to clipboard
Yuxia Wang, Revanth Gangi Reddy, Zain Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate naturally sounding answers over a broad range of human inquiries, but they often generate answers that contradict real-world facts. |
| Approach: | They propose a framework for annotating and evaluating the factuality of large language models . they propose 'factcheck-bench' which provides a multi-stage annotation scheme . |
| Outcome: | The proposed framework outperforms several popular LLM fact-checkers in claim, sentence, and document levels. |
TAPASGO: Transfer Learning towards a German-Language Tabular Question Answering Model (2024.lrec-main)
Copied to clipboard
| Challenge: | This paper examines the performance and limitations of fine-tuned models for tabular data analysis. |
| Approach: | They propose to use fine-tuning mechanics to fine- tune an English model towards a potential German model for tabular data analysis. |
| Outcome: | The proposed model outperforms the original model on German training data. |
UkraiNER: A New Corpus and Annotation Scheme towards Comprehensive Entity Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Named entity recognition excludes nested, discontinuous, non-named entities in practice . despite attempts to broaden their coverage, the most restrictive variant of NER remains the default . |
| Approach: | They propose a new annotation scheme that offers higher comprehensiveness while preserving simplicity. |
| Outcome: | The proposed scheme offers higher comprehensiveness while preserving simplicity . it also includes an annotation tool to implement the scheme on the corpus UkraiNER . |